Repository navigation
FEAT Configure scan targets before launch - #3926
Richard Lundeen (richlundeen) wants to merge 10 commits into
Conversation
Decouple scenario discovery from runtime roles, validate selected dependencies before admission, and preserve explicit setup and saved identities across Python, API, CLI, GUI, and suites. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Explicit scoring targets can be silently ignored, standard seeded tests fail without an optional environment variable, and readiness checks miss target requirements and changes to execution dependencies.
7 open issues
Default screenshot output when the evidence directory is unset · New Honor explicit scoring targets with registered default scorers · New Keep the scoring target picker visible for invalid selections · New Align readiness factory resolution with scenario execution · New Include selected additive converters in readiness hashes · New Validate helper target requirements during readiness · New Validate and bind AdversarialBenchmark generation targets · New
What changed in this PR
This PR makes scan dependencies explicit across PyRIT’s Python, CLI, API, and GUI entry points, removing implicit OpenAI helper construction while retaining deliberately configured defaults.
Changes:
- Resolve scoring and generation dependencies lazily, with readiness checks before execution.
- Add suite preflight, saved dependency references, and in-place target/scorer setup.
- Update regression coverage and migration examples for explicit role configuration.
| File | Description |
|---|---|
| tests/unit/setup/test_technique_initializer.py | Verify discovery avoids implicit provider construction. |
| tests/unit/setup/test_load_default_datasets.py | Check provider-free scenario metadata discovery. |
| tests/unit/scenario/scenarios/adaptive/test_text_adaptive.py | Update explicit helpers and resume expectations. |
| tests/unit/scenario/garak/test_latent_injection.py | Update dependency-change error expectation. |
| tests/unit/scenario/garak/test_figstep.py | Assert retained scorer configuration. |
| tests/unit/scenario/garak/test_audio_achilles_heel.py | Assert retained scorer configuration. |
| tests/unit/scenario/foundry/test_red_team_agent.py | Cover deferred scorer setup and explicit helpers. |
| tests/unit/scenario/core/test_scenario.py | Account for persisted role references. |
| tests/unit/scenario/core/test_scenario_target_defaults.py | Verify configured defaults replace provider fallbacks. |
| tests/unit/scenario/core/test_scenario_readiness.py | Add role readiness and dependency-drift tests. |
| tests/unit/scenario/benchmark/test_adversarial.py | Update lazy scorer expectations. |
| tests/unit/scenario/airt/test_scam.py | Update explicit helpers and deferred configuration. |
| tests/unit/scenario/airt/test_psychosocial.py | Verify sub-harm scorers remain lazy. |
| tests/unit/scenario/airt/test_leakage.py | Update target identity and lazy scorer assertions. |
| tests/unit/scenario/airt/test_jailbreak.py | Update lazy scorer identity expectation. |
| tests/unit/scenario/airt/test_cyber.py | Update lazy scorer identity expectation. |
| tests/unit/models/test_scenario_suite_execution.py | Distinguish preview and launch target requirements. |
| tests/unit/cli/test_pyrit_scan.py | Check helper-role flag parity with the shell. |
| tests/unit/backend/test_target_service.py | Test explicit connection-field validation. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Cover rejection before suite child admission. |
| tests/unit/backend/test_scenario_service.py | Update lazy helper-resolution expectations. |
| tests/unit/backend/test_scenario_run_service.py | Cover modern role references and initialization metadata. |
| tests/end_to_end/browser_initializer.py | Register offline targets, presets, and suites. |
| pyrit/scenario/scenarios/technique/execute.py | Defer default scorer resolution. |
| pyrit/scenario/scenarios/garak/propile.py | Use the resolved scorer directly. |
| pyrit/scenario/scenarios/garak/prompt_inject.py | Use the resolved scorer directly. |
| pyrit/scenario/scenarios/garak/figstep.py | Build scoring configuration during attack construction. |
| pyrit/scenario/scenarios/garak/doctor.py | Defer default scorer resolution. |
| pyrit/scenario/scenarios/garak/divergence.py | Use the resolved scorer directly. |
| pyrit/scenario/scenarios/garak/audio_achilles_heel.py | Defer attack scoring configuration. |
| pyrit/scenario/scenarios/garak/api_key.py | Simplify resolved scorer access. |
| pyrit/scenario/scenarios/foundry/red_team_agent.py | Resolve helpers according to selected compositions. |
| pyrit/scenario/scenarios/benchmark/adversarial.py | Defer default scorer resolution. |
| pyrit/scenario/scenarios/airt/scam.py | Defer scoring and generation configuration. |
| pyrit/scenario/scenarios/airt/rapid_response.py | Defer default scorer resolution. |
| pyrit/scenario/scenarios/airt/psychosocial.py | Lazily prepare and bind selected sub-harm scorers. |
| pyrit/scenario/scenarios/airt/multilingual.py | Preserve explicit generation targets and defer scoring. |
| pyrit/scenario/scenarios/airt/leakage.py | Defer default scorer resolution. |
| pyrit/scenario/scenarios/airt/jailbreak.py | Defer default scorer resolution. |
| pyrit/scenario/scenarios/airt/cyber.py | Defer default scorer resolution. |
| pyrit/scenario/scenarios/adaptive/adaptive_scenario.py | Expose adaptive factories to readiness checks. |
| pyrit/scenario/core/scenario.py | Add lazy roles, readiness checks, and dependency references. |
| pyrit/scenario/core/scenario_target_defaults.py | Remove implicit OpenAI helper fallbacks. |
| pyrit/registry/scenario_suite_resolver.py | Forward explicit scoring roles through suite resolution. |
| pyrit/registry/components/scenario_registry.py | Add pre-initialization configuration validation callback. |
| pyrit/models/scenario_suite_execution.py | Separate suite preview and launch requests. |
| pyrit/models/scenario_readiness.py | Define shared readiness, binding, and issue models. |
| pyrit/models/catalog/scenario.py | Add explicit scoring-role request fields. |
| pyrit/models/__init__.py | Export the suite readiness request lazily. |
| pyrit/cli/pyrit_shell.py | Forward explicit helper-target selections. |
| pyrit/cli/pyrit_scan.py | Add scoring and adversarial target flags. |
| pyrit/cli/_cli_args.py | Parse helper-target flags in shell commands. |
| pyrit/backend/services/target_service.py | Require metadata-declared connection fields. |
| pyrit/backend/services/scorer_service.py | Expose objective-scoring compatibility. |
| pyrit/backend/services/scenario_suite_execution_service.py | Preflight children and carry dependency references. |
| pyrit/backend/services/scenario_service.py | Forward scoring targets into scenario configuration. |
| pyrit/backend/services/scenario_run_service.py | Validate and persist effective launch dependencies. |
| pyrit/backend/services/scenario_configuration_resolver.py | Resolve named roles and preview readiness. |
| pyrit/backend/routes/scenarios.py | Add scenario readiness endpoint. |
| pyrit/backend/routes/scenario_suites.py | Add suite readiness endpoint. |
| pyrit/backend/models/scorers.py | Add objective-scoring support metadata. |
| frontend/src/utils/targetIdentity.ts | Label resolved target-role bindings. |
| frontend/src/types/index.ts | Add readiness and explicit scoring-role types. |
| frontend/src/services/api.ts | Add readiness and scorer-creation API calls. |
| frontend/src/components/Scorers/CreateScorerDialog.tsx | Add metadata-driven objective scorer creation. |
| frontend/src/components/Scorers/CreateScorerDialog.test.tsx | Cover scorer creation and failure states. |
| frontend/src/components/ScenarioSuites/LaunchSuiteDialog.tsx | Add readiness gating and in-place role setup. |
| frontend/src/components/ScenarioSuites/LaunchSuiteDialog.test.tsx | Test readiness-gated suite launch. |
| frontend/src/components/ScenarioSuites/LaunchSuiteDialog.styles.ts | Prevent dialog content from shrinking. |
| frontend/src/components/Scenarios/ScenarioDetail.tsx | Add explicit role setup and readiness-gated launch. |
| frontend/src/components/Scenarios/ScenarioDetail.test.tsx | Cover readiness errors and stale responses. |
| frontend/src/components/helpDefinitions.ts | Clarify explicit generation-role selection. |
| frontend/src/components/Config/CreateTargetDialog.tsx | Return created targets and honor endpoint metadata. |
| frontend/src/components/Config/CreateTargetDialog.test.tsx | Cover optional endpoint configuration. |
| frontend/src/components/Chat/ChatScorerPicker.tsx | Support resolved default-scorer labels. |
| frontend/e2e/explicit-targets.seeded.spec.ts | Add offline scan and suite setup acceptance flows. |
| doc/scanner/0_scanner.md | Document explicit helper-target CLI flags. |
| doc/code/setup/2_resiliency.py | Configure scoring explicitly in the retry example. |
| doc/code/setup/2_resiliency.ipynb | Mirror explicit scoring setup and annotate historical output. |
| doc/code/setup/1_configuration.py | Explain deliberate environment-based initialization. |
| doc/code/setup/1_configuration.ipynb | Mirror configuration migration guidance. |
| doc/code/setup/0_setup.md | Document role configuration, readiness, and migration. |
| doc/code/scenarios/2_custom_scenario_parameters.py | Update scenario import and explain role prerequisites. |
| doc/code/scenarios/2_custom_scenario_parameters.ipynb | Mirror import and explanatory updates. |
Review effort: Balanced
Preserve explicit API connection-field validation in the imported HuggingFace event-loop regression by supplying its test-only token. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bind benchmark and converter dependencies, align effective factories and actual consumer capabilities, preserve scorer rubrics, and repair launch correction/confirmation UX. Cover frozen suite references at real registry admission. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retain positive reuse and deliberate fresh edit-reruns, assert earlier drift rejection before datasets or new results, and configure the helper explicitly when constructing synthetic old saved runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Check every actual shared-helper consumer before work, including accepted simulated seeds with uses_adversarial=False. Preserve native helper success, objective-incompatible SKIP behavior and configured translation adaptation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
In response to this review. Addressed the seven causes summarized here in 962b86d: seeded output fallback, explicit scoring selection, recoverable scoring controls, factory parity, converter identities, actual helper capabilities and benchmark-owned roles. This includes the subsequently identified simulation bypass when the outer adversarial flag is explicitly false. The repair also adds the missing nonempty suite admission regression and groups repeated suite remediation. Local role-confirmation/model-help observations remain separately framed. Confirmed at final published head 1e56308. The subsequent change is seeded-test-only: deferred provenance fixture setup and test-owned registry/explicit offline execution assertions. The full normal seeded shard passed (112 tests, evidence directory unset). Prior source fixes and their recorded visual evidence are unchanged; those earlier visual checks were not rerun. |
Defer packaged compatibility reads until fixture execution. Compare complete target registry identity deltas and persisted offline run dependencies instead of prohibiting other specs' explicit OpenAI-shaped registrations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.
Keep setup focused on selecting only needed scan roles and preserving environment-based initialization. Remove implementation and review-detail exposition. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve explicit scan roles through launch-form extraction and scenario run guards. Update readiness consumer mocks without changing production fallbacks or presentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Update paired documentation, suite contracts and readiness fixtures to the public import policy without behavior or notebook output changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Assert disabled launch and actionable missing-target readiness before selecting the objective target. Preserve target-free estimates, direct API rejection and exact dataset execution checks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.
|
Superseded by #3794, which merged with these changes. |


Description
Let users configure a scan before connecting a model. Browsing scenarios and selecting techniques no longer requires
OPENAI_CHAT_MODELor creates an OpenAI target behind the scenes.Scanner with
OPENAI_CHAT_MODELunsetBoth screenshots are real production builds with the variable unset and no configured targets or scorers. Before is the pre-change baseline (
80f28af); after is this PR (99d7997). No scan or provider request was submitted.